Search and knowledge base question answering for a voice user interface

ABSTRACT

A voice-controlled question answering system that is capable of answering questions using both a knowledge base and a search engine. The knowledge base is used to answer questions when answers to those questions are contained in the knowledge base. If an answer using the knowledge base is unavailable, and if the question is suitable for answering using an unstructured search approach, the system may obtain an answer using a search engine. The search engine results may be processed to obtain an answer to the question suitable for output using a voice user interface.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of, and claims the benefit ofpriority of, U.S. Non-provisional patent application Ser. No.16/444,741, filed Jun. 18, 2019 and entitled “SEARCH AND KNOWLEDGE BASEQUESTION ANSWERING FOR A VOICE USER INTERFACE”, and scheduled to issueas U.S. Pat. No. 10,642,577, which is a continuation of, and claims thebenefit of priority of, U.S. Non-provisional patent application Ser. No.15/707,496, filed Sep. 18, 2017 and entitled “SEARCH AND KNOWLEDGE BASEQUESTION ANSWERING FOR A VOICE USER INTERFACE,” issued as U.S. Pat. No.10,331,402, which claims priority to provisional U.S. Patent ApplicationNo. 62/512,359, entitled “SEARCH AND KNOWLEDGE BASE QUESTION ANSWERINGFOR A VOICE USER INTERFACE” filed on May 30, 2017 in the names of DanielLewis Spector, et al. The contents of each of which is expresslyincorporated herein by reference in their entireties.

BACKGROUND

Speech recognition systems have progressed to the point where humans caninteract with computing devices entirely relying on speech. Such systemsemploy techniques to identify the words spoken by a human user based onthe received audio input. Speech recognition combined with naturallanguage understanding processing techniques enable speech-based usercontrol of a computing device to perform tasks based on the user'sspoken commands. Speech recognition may also include converting a user'sspeech into text data which may then be provided to various text-basedsoftware applications.

Speech processing may be used by computers, hand-held devices, telephonecomputer systems, kiosks, and a wide variety of other devices to improvehuman-computer interactions.

BRIEF DESCRIPTION OF DRAWINGS

For a more complete understanding of the present disclosure, referenceis now made to the following description taken in conjunction with theaccompanying drawings.

FIG. 1 illustrates a quasi-semantic question answering system accordingto embodiments of the present disclosure.

FIG. 2A is a diagram of components of a system according to embodimentsof the present disclosure.

FIG. 2B is a diagram of components of a system for performing naturallanguage understanding according to embodiments of the presentdisclosure.

FIG. 3 illustrates obtaining content from a structured knowledge baseand other data sources according to embodiments of the presentdisclosure.

FIG. 4 illustrates obtaining search-based results for a user queryreceived from a voice user interface according to embodiments of thepresent disclosure.

FIGS. 5A-5B are a flow chart illustrating search-based questionanswering according to embodiments of the present disclosure.

FIG. 6 is a block diagram conceptually illustrating example componentsof a device according to embodiments of the present disclosure.

FIG. 7 is a block diagram conceptually illustrating example componentsof a server according to embodiments of the present disclosure.

FIG. 8 illustrates an example of a computer network for use with thesystem.

DETAILED DESCRIPTION

Automatic speech recognition (ASR) is a field of computer science,artificial intelligence, and linguistics concerned with transformingaudio data associated with speech into text representative of thatspeech. Similarly, natural language understanding (NLU) is a field ofcomputer science, artificial intelligence, and linguistics concernedwith enabling computers to derive meaning from text input containingnatural language. ASR and NLU are often used together as part of aspeech processing system. The combination of ASR and NLU may be referredto herein as speech processing.

ASR and NLU can be computationally expensive. That is, significantcomputing resources may be needed to process ASR and NLU processingwithin a reasonable time frame. Because of this, a distributed computingenvironment may be used when performing speech processing. An example ofsuch a distributed environment may involve a local device having one ormore microphones being configured to capture sounds from a user speakingand circuitry configured to convert those sounds into an audio signal.The audio signal/data may then be sent to a downstream remote device forfurther processing, such as for converting the audio signal into anultimate command. The command may then be executed by a remote and/or(the) local device(s) depending on the command itself.

In certain situations, a user utterance may include a question to beanswered by the system. That is, the user may ask the system to providesome information in response to the utterance. The ultimate command inthis example would be to provide the information requested. As part ofNLU processing, in order to respond to a user's query, the system mayrely on a structured knowledge base to obtain or confirm informationrequested by the query. A structured knowledge base is a data store thatincludes facts organized in a particular manner that may be used torespond to user queries.

In one example, a knowledge base may include a collection of tuples andmay encode formal semantics on the data stored within the knowledgebase. In such a knowledge base certain knowledge bases or informationstores may store information in a “tuple” format, where entities arejoined together in pairs, triplets, or the like, that define therelationship between two things. For example a tuple such as [John Doe,husband of, Jane Doe] may describe that John Doe is Jane Doe's husband.Many such tuples may exist in a knowledge base and may be usable toanswer incoming informational queries. For example, the above tuple maybe used to answer a question such as “who is Jane Doe's husband,” “whois John Doe married to” or the like.

One drawback to a tuple-based approach, or other knowledge base that maystore information in a limited format, is that it may be difficult forthe system to infer knowledge between different tuples. For example, ifother tuples existed such as [John Doe, date of marriage, 2002] and[Jake Doe, brother of, John Doe], a tuple-based system may not be ableto answer a question such as “when did John Doe become thebrother-in-law of Jane Doe” as the system would need to traverse andlink several tuples regarding the various relationships and dates toobtain the needed information. Such processing would be difficult for atuple based system.

In another structured knowledge base information may be stored in a formrepresentative of a knowledge graph, which may comprise a directedacyclic graph (DAG) which graphs nodes representing entities (e.g.,people, places, things) connected by vertices or links where each linkcorresponds to a relationship. To answer questions about particularentities in a knowledge graph, the system may traverse a path along theknowledge graph from one entity to another and track the informationalong the traversed links to obtain information about how one entityrelates to another. For example, a DAG may include a node for Jane Doe,a node for John Doe, a node for Jake Doe, etc. A link between Jane Doeand John Doe may be associated with a variety of information such as“married in 2002,” “met in 2000,” etc. A link between John Doe and JakeDoe may be associated with information such as “brothers” or the like.Even if there is no direct link between the node for Jane Doe and thenode for Jake Doe, the system may be able to determine a path betweenthe two, and gather information along the vertices along that path(e.g., from Jane Doe to John Doe and then from John Doe to Jake Doe) toobtain information needed to answer the query “when did John Doe becomethe brother-in-law of Jane Doe.”

The knowledge base may also include a schema (for example defined byclasses and properties) to organize its data.

There are many types of queries that a user may pose from simplesingle-fact questions (for example, “what is the capital of France?”) tomore detailed questions involving time, place, multiple entities, etc.(for example, “when was Tom Cruise married to Nicole Kidman?”). Queryresolution systems may rely on a rule/template based approach. In suchan approach, a series of rules may be used to analyze the incoming textdata of the query to identify the intent of the query (i.e., whatinformation is being sought) and what entities are named in the querythat are needed to provide the desired information. In a rule/templatebased approach, rules may be applied to text data, where the results ofeach rule's processing may be given a confidence score where the ruleresults corresponding to a highest confidence score are selected, parsedand passed on to the knowledge base.

In order to retrieve answer data from a knowledge base, a query mayfirst be parsed and new data generated in a form recognizable by theknowledge base. That data may then be sent to the knowledge base toobtain information responsive to a user query. That responsiveinformation may then be formatted as output data to be returned to auser, either as synthesized speech to speak the answer to the user or insome other form (such as email, text message, visual display userelement, etc.).

One benefit to a structured knowledge base is that such a knowledge basemay result from some level of curation and/or compilation such that theinformation in the structured knowledge base may have a higher level ofreliability and otherwise may generally be more accurate thatinformation that may otherwise be available. Further, it may bedesirable to use a knowledge base to answer questions for a voice userinterface (VUI) where typically only one answer may be returned to auser as opposed to for a graphical user interface (GUI) where multiplepotential answer may be displayed to a user, allowing the user to selectwhich answer most closely corresponds to the information the user wasseeking.

One drawback to using a knowledge base is that not all questions will beanswerable using the knowledge base, either because the knowledge basedoes not include sufficient information to answer the question, orbecause the query may be phrased in a way that makes it difficult fornatural language components to understand what information the user isrequesting. Specifically, in order to form a query to the knowledge basethe system may perform NLU processing to make a sematic understanding ofthe question and the information being sought, so the system canformulate the query to the knowledge base to obtain the desiredquestion. Enabling a computing system to undertake a semanticunderstanding of every potential question would be technicallychallenging.

An alternative to a knowledge base query is a web query, where a searchstring is input into a search engine. However the result for such webqueries often take the form of a list of Internet links rather than ananswer to a specific question, and further web queries make no effort atany semantic understanding, relying instead on a solely keyword basedsearch approach. Further, as noted above such general search queriesoften obtain multiple potential responsive results, but a VUI may onlybe able to return a single result.

Offered is a question answering system that incorporates a knowledgebase but also allows the ability to use a web search to obtain resultsfor a spoken query, particularly if the knowledge base is unable toprovide an answer. A variety of components operate to ensure that aquestion is appropriate for obtaining an answer using a web search aswell as to ensure that a search result is appropriate for ultimatelyreturning to a user. The system can process search results to obtain atop sentence or two (or other portion of text) corresponding to ananswer to the user's query, which may ultimately be returned through aVUI.

This combined approach of answering questions allows the system toanswer both questions that call for information in, and are parse-ableinto a form understood by, a knowledge base (for example “what is thecapital of France?” or “who was the first President of the UnitedStates?”) as well as more complex questions that are difficult to parseor whose answers may not be in a knowledge base but may be found on theInternet or in other sources (for example “how many black keys are therein a piano?” or “who was the American music promoter who brought theBeatles to play in Shea Stadium?”).

FIG. 1 shows a system 100 configured to use a quasi-search basedapproach to question answering. Although FIG. 1, and furtherfigures/discussion, illustrate the operation of the system in aparticular order, the steps described may be performed in a differentorder (as well as certain steps removed or added) without departing fromthe intent of the disclosure. As shown in FIG. 1, a system 100 mayinclude one or more devices 110 local to user(s) 10, as well as one ormore networks 199 and one or more servers 120 connected to device 110across network(s) 199. The server(s) 120 (which may be one or moredifferent physical devices) may be capable of performing traditionalspeech processing (such as ASR, NLU, question parsing, etc.) asdescribed herein. A single server may be capable of performing allspeech processing or multiple server(s) 120 may combine to perform thespeech processing. Further, the server(s) 120 may be configured toexecute certain commands, such as answering queries spoken by user 10.In addition, certain speech detection or command execution functions maybe performed by device 110.

As shown in FIG. 1, at runtime a user 10 may speak an utterance(represented by input audio 11) including a question to a local device110. The device 110 may convert the audio 11 into audio data 111 andsend the audio data to the server(s) 120. The server(s) 120 may thenreceive (140) the input audio data and perform (142) speech processingon the audio data to obtain text (through ASR processing) and/or NLUresults that determine the utterance included a question. The server(s)120 may then determine (144) whether the question is appropriate for aweb-search answering approach using one or more components as explainedbelow. For example, the system may use one or more trained machinelearning models to determine whether the question is appropriate. Thesystem may then send (146) portions of the question that include thetext of the user's query to a search engine for processing. The systemmay then receive (148) search results from the search engine. The searchresults may include links, such as uniform resource locators (URLs) orother pointers to data source(s) 180 that may include informationresponsive to the query. The system may access and obtain (150) searchresult data pointed to by the links and process the data from the datasource(s) 180 that are potentially responsive to the user's question.The system may then determine a sentence or other portion of data fromthe search result data to determine (152) answer data from the topsearch result. The system may then combine (154) the answer data with asource indicator (such as text indicating what website or other datasource(s) the answer data was obtained from). The system may thenperform (156) TTS on output data to create output audio data speakingthe answer obtained from the web search along with an indicator of thesource of the answer. For example, the system may respond to the user'squery with synthesized speech such as “I found an answer to yourquestion. According to [website X]: [answer data].” As explained below,the system may also attempt to answer the user's query using astructured knowledge base 190 and may select from answer data obtainedfrom the structured knowledge base 190 or from the search processdescribed herein.

The system may operate using various components as described in FIG. 2A.The various components illustrated FIG. 2A may be located on a same ordifferent physical devices. Communication between various componentsillustrated in FIG. 2A may occur directly or across a network(s) 199.

An audio capture component, such as a microphone or array of microphonesof the device 110 a or other device, captures the input audio 11corresponding to a spoken utterance. The device 110 a, using a wakeworddetection component 220, processes audio data corresponding to the inputaudio 11 to determine if a keyword (e.g., a wakeword) is detected in theaudio data. Following detection of a wakeword, the device 110 a sendsaudio data 211, corresponding to the utterance, to a server(s) 120 forprocessing.

Upon receipt by the server(s) 120, the audio data 211 may be sent to anorchestrator component 230. The orchestrator component 230 may includememory and logic that enables the orchestrator component 230 to transmitvarious pieces and forms of data to various components of the system.

The orchestrator component 230 sends the audio data 211 to a speechprocessing component. A speech recognition component 250 (which may bepart of the speech processing component) transcribes the audio data 211into one more textual interpretations representing speech contained inthe audio data 211. The speech recognition component 250 interprets thespoken utterance based on a similarity between the spoken utterance andpre-established language models. For example, the speech recognitioncomponent 250 may compare the audio data 211 with models for sounds(e.g., subword units or phonemes) and sequences of sounds to identifywords that match the sequence of sounds spoken in the utterancerepresented in the audio data 211.

Alternatively, a keyboard (e.g., presented on a touch sensitive display)of a device 110 b may capture textual input corresponding to a usercommand. The device 110 b may generate text data corresponding to theinput text, and may send the text data to the server(s) 120. Uponreceive by the server(s) 120, the text data may be sent to theorchestrator component 230.

If the user command was received by the server(s) 120 as audio data fromthe device 110 a, the speech recognition component 250 sends text datagenerated using speech recognition processing to the orchestrator, whichthen sends the text data to the query handling component 255. The textdata sent from the speech recognition component 250 may include a topscoring textual interpretation of the spoken utterance or may include anN-best list including a group of textual interpretations and potentiallytheir respective scores.

The orchestrator component 230 sends the text data (either generated bythe ASR component 250 or otherwise) to a query handling component 255.The query handling component may attempt to process the text datacorresponding to a user query in a number of ways. Those ways may beexecuted at least partially in parallel. The text data may be sent toquestion and answer (Q&A) service component 265 which may attempt toprocess the text data, where the Q&A service component 265 assumes thatthe text data corresponds to an information request. As the Q&A servicecomponent 265 may operate at least partially in parallel to a naturallanguage understanding (NLU) component 260, the Q&A service component265 may not know if the text data actually corresponds to an informationrequest, but processing the text data at least partially in parallelbetween the NLU component 260 and Q&A service component 265 may resultin reduced user latency.

The text data may be sent to a knowledge base (KB) query service whichmay work in conjunction with a structured knowledge base 279 to attemptto obtain information responsive to the text data. As noted above, theKB 279 may include a collection of tuples or otherwise semantically (orotherwise) encoded data that can be queried to obtain information. Oneexample of such a knowledge base is Amazon's EVI knowledge base.

At least partially in parallel to the KB query service 275 attempting tofind an answer to the text data using KB 279, the system may operate asearch manager 410 to attempt to find an answer to the text data usingan unstructured web search, as discussed below.

At least partially in parallel to the operations of the Q&A servicecomponent 265, the system may process the text data using the NLUcomponent 260. The NLU component 260 attempts to make a semanticinterpretation of the phrases or statements represented in the textdata. That is, the NLU component 260 determines one or more meaningsassociated with the phrases or statements represented in the text databased on individual words represented in the text data. The NLUcomponent 260 interprets a text string to derive an intent of the usercommand represented in the text data (e.g., an action that the userdesires be performed) as well as pertinent pieces of information in thetext data that allow a device (e.g., the device 110 a, the device 110 g,the server(s) 120, the application server(s) 125, etc.) to complete theintent. For example, if the text data corresponds to “call mom,” the NLUcomponent 260 may determine the user intended to activate a telephone inhis/her device and to initiate a call with a contact matching the entity“mom.”

Following ASR processing, the ASR results may be sent by the AScomponent 250 to other processing components, which may be local to thedevice performing ASR and/or distributed across the network(s) 199. Forexample, ASR results in the form of a single textual representation ofthe speech, an N-best list including multiple hypotheses and respectivescores, lattice, etc. may be sent to a server, such as server 120, fornatural language understanding (NLU) processing, such as conversion ofthe text into commands for execution, either by the device 110, by theserver 120, or by another device (such as a server running a specificapplication like a search engine, etc.).

The device performing NLU processing 260 (e.g., server 120) may includevarious components, including potentially dedicated processor(s),memory, storage, etc. For example, as shown in FIG. 2B, a deviceconfigured for NLU processing may include a named entity recognition(NER) module 252 and intent classification (IC) module 264, a resultranking and distribution module 266, and knowledge base 272. The NLUprocess may also utilize gazetteer information (284 a-284 n) stored inentity library storage 282. The gazetteer information may be used forentity resolution, for example matching ASR results with differententities (such as song titles, contact names, etc.) Gazetteers may belinked to users (for example a particular gazetteer may be associatedwith a specific user's music collection), may be linked to certaindomains (such as shopping), or may be organized in a variety of otherways.

The NLU process takes textual input (such as processed from ASR 250based on the utterance 11) and attempts to make a semanticinterpretation of the text. That is, the NLU process determines themeaning behind the text based on the individual words and thenimplements that meaning. NLU processing 260 interprets a text string toderive an intent or a desired action from the user as well as thepertinent pieces of information in the text that allow a device (e.g.,device 110) to complete that action. For example, if a spoken utteranceis processed using ASR 250 and outputs the text “call mom” the NLUprocess may determine that the user intended to activate a telephone inhis/her device and to initiate a call with a contact matching the entity“mom.”

The NLU may process several textual inputs related to the sameutterance. For example, if the ASR 250 outputs N text segments (as partof an N-best list), the NLU may process all N outputs to obtain NLUresults.

The NLU process may be configured to parsed and tagged to annotate textas part of NLU processing. For example, for the text “call mom,” “call”may be tagged as a command (to execute a phone call) and “mom” may betagged as a specific entity and target of the command (and the telephonenumber for the entity corresponding to “mom” stored in a contact listmay be included in the annotated result).

To correctly perform NLU processing of speech input, the NLU process 260may be configured to determine a “domain” of the utterance so as todetermine and narrow down which services offered by the endpoint device(e.g., server 120 or device 110) may be relevant. For example, anendpoint device may offer services relating to interactions with atelephone service, a contact list service, a calendar/schedulingservice, a music player service, etc. Words in a single text query mayimplicate more than one service, and some services may be functionallylinked (e.g., both a telephone service and a calendar service mayutilize data from the contact list).

The name entity recognition module 262 receives a query in the form ofASR results and attempts to identify relevant grammars and lexicalinformation that may be used to construe meaning. To do so, a nameentity recognition module 262 may begin by identifying potential domainsthat may relate to the received query. The NLU storage 272 includes adatabases of devices (274 a-274 n) identifying domains associated withspecific devices. For example, the device 110 may be associated withdomains for music, telephony, calendaring, contact lists, anddevice-specific communications, but not video. In addition, the entitylibrary may include database entries about specific services on aspecific device, either indexed by Device ID, User ID, or Household ID,or some other indicator.

A domain may represent a discrete set of activities having a commontheme, such as “shopping”, “music”, “calendaring”, etc. As such, eachdomain may be associated with a particular language model and/or grammardatabase (276 a-276 n), a particular set of intents/actions (278 a-278n), and a particular personalized lexicon (286). Each gazetteer (284a-284 n) may include domain-indexed lexical information associated witha particular user and/or device. For example, the Gazetteer A (284 a)includes domain-index lexical information 286 aa to 286 an. A user'smusic-domain lexical information might include album titles, artistnames, and song names, for example, whereas a user's contact-listlexical information might include the names of contacts. Since everyuser's music collection and contact list is presumably different, thispersonalized information improves entity resolution.

A query is processed applying the rules, models, and informationapplicable to each identified domain. For example, if a querypotentially implicates both communications and music, the query will beNLU processed using the grammar models and lexical information forcommunications, and will be processed using the grammar models andlexical information for music. The responses based on the query producedby each set of models is scored (discussed further below), with theoverall highest ranked result from all applied domains is ordinarilyselected to be the correct result.

An intent classification (IC) module 264 parses the query to determinean intent or intents for each identified domain, where the intentcorresponds to the action to be performed that is responsive to thequery. Each domain is associated with a database (278 a-278 n) of wordslinked to intents. For example, a music intent database may link wordsand phrases such as “quiet,” “volume off,” and “mute” to a “mute”intent. The IC module 264 identifies potential intents for eachidentified domain by comparing words in the query to the words andphrases in the intents database 278.

In order to generate a particular interpreted response, the NER 262applies the grammar models and lexical information associated with therespective domain. Each grammar model 276 includes the names of entities(i.e., nouns) commonly found in speech about the particular domain(i.e., generic terms), whereas the lexical information 286 from thegazetteer 284 is personalized to the user(s) and/or the device. Forinstance, a grammar model associated with the shopping domain mayinclude a database of words commonly used when people discuss shopping.

The intents identified by the IC module 264 are linked todomain-specific grammar frameworks (included in 276) with “slots” or“fields” to be filled. For example, if “play music” is an identifiedintent, a grammar (276) framework or frameworks may correspond tosentence structures such as “Play {Artist Name},” “Play {Album Name},”“Play {Song name},” “Play {Song name} by {Artist Name},” etc. However,to make recognition more flexible, these frameworks would ordinarily notbe structured as sentences, but rather based on associating slots withgrammatical tags.

For example, the NER module 260 may parse the query to identify words assubject, object, verb, preposition, etc., based on grammar rules andmodels, prior to recognizing named entities. The identified verb may beused by the IC module 264 to identify intent, which is then used by theNER module 262 to identify frameworks. A framework for an intent of“play” may specify a list of slots/fields applicable to play theidentified “object” and any object modifier (e.g., a prepositionalphrase), such as {Artist Name}, {Album Name}, {Song name}, etc. The NERmodule 260 then searches the corresponding fields in the domain-specificand personalized lexicon(s), attempting to match words and phrases inthe query tagged as a grammatical object or object modifier with thoseidentified in the database(s).

This process includes semantic tagging, which is the labeling of a wordor combination of words according to their type/semantic meaning.Parsing may be performed using heuristic grammar rules, or an NER modelmay be constructed using techniques such as hidden Markov models,maximum entropy models, log linear models, conditional random fields(CRF), and the like.

For instance, a query of “play mother's little helper by the rollingstones” might be parsed and tagged as {Verb}: “Play,” {Object}:“mother's little helper,” {Object Preposition}: “by,” and {ObjectModifier}: “the rolling stones.” At this point in the process, “Play” isidentified as a verb based on a word database associated with the musicdomain, which the IC module 264 will determine corresponds to the “playmusic” intent. No determination has been made as to the meaning of“mother's little helper” and “the rolling stones,” but based on grammarrules and models, it is determined that these phrase relate to thegrammatical object of the query.

The frameworks linked to the intent are then used to determine whatdatabase fields should be searched to determine the meaning of thesephrases, such as searching a user's gazette for similarity with theframework slots. So a framework for “play music intent” might indicateto attempt to resolve the identified object based {Artist Name}, {AlbumName}, and {Song name}, and another framework for the same intent mightindicate to attempt to resolve the object modifier based on {ArtistName}, and resolve the object based on {Album Name} and {Song Name}linked to the identified {Artist Name}. If the search of the gazetteerdoes not resolve the a slot/field using gazetteer information, the NERmodule 262 may search the database of generic words associated with thedomain (in the NLU's knowledge base 272). So for instance, if the querywas “play songs by the rolling stones,” after failing to determine analbum name or song name called “songs” by “the rolling stones,” the NER262 may search the domain vocabulary for the word “songs.” In thealternative, generic words may be checked before the gazetteerinformation, or both may be tried, potentially producing two differentresults.

The comparison process used by the NER module 262 may classify (i.e.,score) how closely a database entry compares to a tagged query word orphrase, how closely the grammatical structure of the query correspondsto the applied grammatical framework, and based on whether the databaseindicates a relationship between an entry and information identified tofill other slots of the framework.

The NER modules 262 may also use contextual operational rules to fillslots. For example, if a user had previously requested to pause aparticular song and thereafter requested that the voice-controlleddevice to “please un-pause my music,” the NER module 262 may apply aninference-based rule to fill a slot associated with the name of the songthat the user currently wishes to play—namely the song that was playingat the time that the user requested to pause the music.

The results of NLU processing may be tagged to attribute meaning to thequery. So, for instance, “play mother's little helper by the rollingstones” might produce a result of: {domain} Music, {intent} Play Music,{artist name} “rolling stones,” {media type} SONG, and {song title}“mother's little helper.” As another example, “play songs by the rollingstones” might produce: {domain} Music, {intent} Play Music, {artistname} “rolling stones,” and {media type} SONG.

The orchestrator component 230 may send output from the NLU component260 (e.g., text data including tags attributing meaning to the words andphrases represented in the text data), and optionally output from theuser recognition component 295 and/or data from the user profile storage270, to one or more applications 290, which may be located on a same orseparate server 120 as part of system 100. FIG. 2A illustrates variousapplications 290 maintained and/or operated by the server(s) 120.However, it should be appreciated that the data sent to the applications290 may also be sent to application servers 125 executing applications.

The destination application 290 may be determined based on the NLUoutput. For example, if the NLU output includes a command to play music,the destination application 290 may be a music playing application, suchas one located on device 110 or in a music playing appliance, configuredto execute a music playing command.

An “application,” as used herein, may be considered synonymous with askill. A “skill” may correspond to a domain and may be software runningon a server(s) 120 that is akin to an application. That is, a skill mayenable a server(s) 120 or application server(s) 125 to execute specificfunctionality in order to provide data or produce some other outputrequested by a user. The system may be configured with more than oneskill. For example a weather service skill may enable the server(s) 120to execute a command with respect to a weather service server(s) 125, acar service skill may enable the server(s) 120 to execute a command withrespect to a taxi or ride sharing service server(s) 125, an order pizzaskill may enable the server(s) 120 to execute a command with respect toa restaurant server(s) 125, etc.

The application 290 to which the orchestrator component 230 sends datamay be chosen based on the output of the natural language component 260.In an example, if the NLU component 260 outputs text data associatedwith an intent to play music, the application 290 selected maycorrespond to a music playing application. In another example, if theNLU component 260 outputs text data associated with an intent to outputweather information, the application 290 selected may correspond to aweather application.

The orchestrator component 230 may send text data output by theapplication 290 to a TTS component 280. The TTS component 280 maysynthesize speech corresponding to received text data. Audio datasynthesized by the TTS component 280 may be sent to the device 110 a (oranother device including a speaker) for output to a user.

An application 290 may output text data to the server(s) 120. In certainconfigurations device 110 is a speech controlled device whose primaryinput/output is speech, either from the user to provide inputs/queriesto the system, or by the system to respond to the user (such asanswering a query) in the form of a synthesized speech output. Suchspoken interactions may be common, for example, with headless devicesthat lack a keyboard, touchscreen, or other input/output mechanisms,though speech control may be used with many different device types.

To create output speech audio (for headless devices or otherwise), thesystem may be configured with a text-to-speech (TTS) component 280. TheTTS component 280 may perform speech synthesis using one or moredifferent methods. In one method of synthesis called unit selection, theTTS component 280 matches text data or a derivative thereof against adatabase of recorded speech. Matching units are selected andconcatenated together to form audio data. In another method of synthesiscalled parametric synthesis, parameters such as frequency, volume, andnoise are varied by the TTS component 280 to create an artificial speechwaveform output. Parametric synthesis uses a computerized voicegenerator, sometimes called a vocoder.

The server(s) 120 may include a user recognition component 295. The userrecognition component 295 may take as input the audio data 211, the textdata received by the server(s) 120 from the device 110 b, and/or thetext data output by the speech recognition component 250. The userrecognition component 295 determines scores indicating whether thecommand represented in the audio data 211 or the text data provided bythe device 110 b originated from particular users. For example, a firstscore may indicate a likelihood that the command originated from a firstuser, a second score may indicate a likelihood that the commandoriginated from a second user, etc. The user recognition component 295also determines an overall confidence regarding the accuracy of userrecognition operations. If the user command is received as audio data211, user recognition may involve comparing speech characteristics inthe audio data 211 to stored speech characteristics of users. Userrecognition may also involve comparing biometric data (e.g., fingerprintdata, iris data, etc.) received by the user recognition component 295 tostored biometric data of users. User recognition may further involvecomparing image data including a representation of at least a feature ofa user with stored image data including representations of features ofusers. Other types of user recognition processes, including those knownin the art, may also or alternatively be used. Output of the userrecognition component 295 may be used to inform natural languageprocessing as well as processing performed by applications 290 (as wellas applications operated by the application server(s) 125).

The server(s) 120 may include a user profile storage 270. The userprofile storage 270 includes data regarding user accounts. Asillustrated, the user profile storage 270 is implemented as part of theserver(s) 120. However, it should be appreciated that the user profilestorage 270 may be located proximate to the server(s) 120, or mayotherwise be in communication with the server(s) 120, for example overthe network(s) 199. The user profile storage 270 may include a varietyof information related to individual users, accounts, etc. that interactwith the system.

A voice-controlled question answering system may make use of all thevarious components discussed above to receive a question from a user,convert that question to audio data, receive and perform ASR on theaudio data to identify text, perform NLU on the text to determine aquestion in the text, formulate a query to the knowledge base using thetext, retrieve an answer from the knowledge base, formulate answer datausing the answer, perform TTS on the answer data to create output audiodata including the answer, send the output audio data to the localdevice and play the output audio data back to a user. It is desirable,as discussed above, to implement a search-based question answeringapproach to supplement the use of the knowledge base to respond to userquestions. Such as supplemental approach will expand the overall abilityof the system to respond to more obscure user questions that areunanswerable by the knowledge base.

As shown in FIG. 3, the system, through server(s) 120 may obtaininformation to respond to a user's query from a structured knowledgebase 190, and/or from one or more data sources 180. In one example thedata sources 180 are available data sources on the internet or otheraccessible storage source. In another example the data sources 180 areother data sources such as a user's private data source (hard drive,music collection, device configurations, etc.) thus enabling the systemto answer queries specific to certain data sources using the techniquesdescribed above.

The system may use components such as those illustrated in FIG. 4 toobtain an answer to a user query using a search based system for a voiceuser interface. As shown the system may include a search manager 410.The search manager 410 may be communicatively connected to a searchengine 480 and one or more data source(s) 180. Although illustrated asconnected over the network 199, the search manager 410, search engine480 and one or more data source(s) 180 may be co-located on one or moreserver(s) 120 or located in various physical arrangements. The searchmanager 410 may include one or more various sub-components such assearch answer controller 420, search acceptability filter 440, searchresult manager 450 and answer relevance classifier 460.

The system may user the components of FIG. 4 to perform a search-basedquery answering component for a voice controlled system, thus providingan alternate question answering service that may operate at runtime toprocess user questions to run along with a knowledge-based service. Asillustrated in FIG. 5A, a system may receive (140) audio datacorresponding to a question. The system may then perform (502) speechrecognition on the audio data to obtain the query text data.Alternatively, the system may receive text data from another source,such as a text-based message being received by the system (like from aquery interface), text data obtained from an interpretive component suchas a component that converts sign language or gesture data into text, orthe like. The text data may then be sent to the query handling component255 for various processing, which may occur serially, in parallel, or atleast partially in parallel. For example, turning to FIG. 5B, the textdata may be sent (523) to an NLU component 260 for NLU processing andmay also be sent (504) to the knowledge base query service 275 forprocessing to determine if the knowledge base can answer the user'squestion.

The NLU component 260 may process the text data to determine (525) NLUresults which may include an indication of an intent corresponding tothe text data. The NLU results may also include indicators of slots,named entities or the like. If the NLU results indicate a Q&A intent(527:Yes) (that is, an intent corresponding to an information requestfor general knowledge or information), the system may then use theoutput determined by the Q&A service component(s) 265. If the NLUresults do not indicate a Q&A intent (527:No), the system will process(529) the NLU results as otherwise called for by the NLU results.

The left hand side of FIG. 5B includes certain steps taken by thecomponents of the Q&A service component(s) 265. As part of theprocessing by the knowledge base query service component 275, the systemmay formulate (524) a query for the knowledge base using the text dataand send (526) the query to the knowledge base 279. If the knowledgebase 279 can answer the question (528:Yes) the system may formulate(530) output answer data using data from the knowledge base. For a voiceuser interface system the system may then perform (156) TTS on theoutput data to obtain output audio data and may send (536) the outputaudio data to a local device so the local device may output speechcorresponding to the answer to the question. If, however, the knowledgebase cannot provide an answer to the question (528:No) the system mayrely on the search-based question answering approach using thecomponents of FIG. 4 and the process described in FIG. 5A.Alternatively, the knowledge base may attempt to answer the user'squestion and may even provide an answer with an assigned confidencescore. The confidence score of the knowledge base answer may be comparedwith a confidence score of an answer provided by using the search-basedapproach (which may occur after such scores are normalized), and theanswer with the highest confidence may be used to answer the user'squestion. Alternatively, if the knowledge base answer does not have aconfidence score above a certain threshold, the system may use an answerobtained using the search-based approach.

The alternate question answering approach illustrated further in FIG. 5Amay be performed at least partially in parallel (i.e., at leastpartially at the same time) to querying the knowledge base to reducelatency in the event a knowledge base answer is unavailable (or is notassociated with a sufficient confidence score) and may be also beperformed at least partially in parallel to the NLU processing by NLUcomponent 260. One difference between querying a knowledge base versussending query text data to a search engine is that the knowledge basequery is typically a structured query configured to align with theoperational parameters of the knowledge base whereas a search using asearch engine is often unstructured, such that the text data is simplyentered into the search engine for the search engine to use whensearching the internet (or other source). As certain query text data maynot be well suited to an unstructured search using a search engine, asshown in FIG. 5A, the system may determine whether the incoming questionis appropriate for the search-based question answering portion of thesystem. As certain kinds of questions may be more suited to being usinga search-based technique (such as questions that omit certain words,questions that are formed as obvious questions, etc.), filtering outless appropriate questions may improve system performance. If a questionis not suitable for answering using a web search or other unstructuredsearch (508:No), processing of the text data by the search manager 410may be discontinued (510) and the text data may be sent or otherwiseindicated for alternate processing, which may include alternatetechniques beyond the knowledge base and/or web search or may involvesimply returning to the user an indication that the system cannot answerthe question if the question is unanswerable by the knowledge base.

To determine if the question is appropriate for answering using a websearch may train one or more machine learning models which may include amulti-class model, classifier, or multiple classifiers that may return ayes or no (or relative score) to indicate whether the incoming text datawill be well handled by the system. The machine learning model(s) usedfor this stage may be trained on a training set of questions that arelabelled either as suitable or not suitable for answering using thesearch-based system. In particular, the model(s) may be trained toidentify questions that may be reliably answered using search-basedtechniques. As further information becomes available about the operationof the search-based question answering, particularly for a voice userinterface (VUI), or as more training data becomes available, themodel(s) may be updated/retrained to ensure proper operation of thesystem. The model(s) may operate on (and be trained on) text data and/oron feature data extracted from the text data. In certain instances,multiple models may be used where each model corresponds to a componentof the search manager 410 (such as each component of the searchacceptability filter 440). If any of the models is used by the system toanalyze the text data and results in a confidence score above a certainthreshold, the system may determine the question is appropriate(508:Yes) for question answering using an unstructured web search.

For example, the text data and/or features obtained from the text data(such as those obtained during step 506 by a feature component 404) maybe sent from a speech processing component to the search manager 410.The text data and/or features may then be sent from the search answercontroller 420 to the search acceptability filter 440. The searchacceptability filter 440 may operate several components that check tosee if the question is appropriate for answering using a search-basedsystem. For example, a blacklist manager 442 may process the text dataand/or features to search the text data for any words that are includedwhich indicate the question is not suitable for search-based answering.Thus, in order for the question to be approved for unstructured searchoperations, the blacklist manager 442 may determine that the words ofthe text data do not include any words in a defined group of unapprovedquestion words. If any such prohibited (i.e., blacklisted) words arefound, the system may determine the question is not suitable (508:No)for search-based answering.

The search acceptability filter 440 may also include a question typemanager 444 that may operate a machine learning model to classify whattype the incoming query is. For example, the question type manager 444may process the text data and/or features to determine if the queryincludes words such as who, what, when, where, to determine what kind ofquestion is being asked and whether that type of question is suitablefor search-based answering. In a particular embodiment, the system mayfocus on specific question words, such as “who” and “what” as thosewords may be particularly well suited for answering using the describedsearch-based approach. The question type manager 444 may determine thatthe question type is in a defined group of approved question types. Ifthe question is determined to not to be in a group of approved questiontypes (e.g., the text data does not include the desired question words),or otherwise does not correspond to a question that will be wellanswered by a search-based, the system may determine the question is notsuitable (508:No) for search-based answering.

The search acceptability filter 440 may also include a suitabilitymanager 446 that may operate a machine learning model (such as a fasttext classifier or other model) that may process the text data and/orother features to determine if the question (for example the form of thequestion) is suitable for search-based answering. The suitabilitymanager 446 may also take as an input (either as part of the featuredata or otherwise) a question type (such as a type determined byquestion type manager 444), question topic (such as a topic or subjectdetermined by question classifier 448) or other data. The model/analysisused by the suitability manager 446 may be customized for thesearch-based answering enabled by the system. For example, in oneparticular embodiment the system may only be configured to be able toanswer certain kinds of questions with reasonable accuracy. Thus thesystem may only determine such questions are suitable. Otherwise thesystem may determine the question is not suitable (508:No) forsearch-based answering.

The suitability manager 446 may output a score corresponding to howsuitable the query text data is for answering using an unstructured websearch. Such a score may correspond to how likely an unstructured websearch (and potential further operations such as those of steps 146-152)is to obtain a correct answer to the question. The suitability manager446 may be trained using training data including many example questionsand whether a correct answer to those questions was obtained using anunstructured web search (and potential further operations such as thoseof steps 146-152). The score output by the suitability manager 446 atruntime may be a binary value (indicating a yes or no) or a numericalvalue, which may be compared to a threshold value to determine if thequestion is suitable for an unstructured search. The threshold value maybe dynamic and/or configurable based on operating conditions of thesystem. For example, the threshold value may be increased if conditionssuggest an unstructured search should not be done under existingcircumstances or the threshold value may be decreased if an unstructuredsearch is desired.

The search acceptability filter 440 may also include a questionclassifier 448 which may classifier the question based on the subjectcorresponding to the question. The question classifier 448 may use acustomized machine learning model or may, for example, use a trainedmodel that is generally used to categorize questions, such as one thatmay also be used to classify incoming queries that may be processedusing the structured knowledge base 190. The question classifier 448 mayprocess the text data and/or features to classify questions by topic(for example questions related to general facts, history, law, etc.).The question classifier 448 may determine that the questiontopic/subject is in a defined group of approved questiontopics/subjects. If the question does not correspond to a topic/subjectthat is well suited for search-based answering, the system may determinethe question is not suitable (508:No) for search-based answering. In oneparticular embodiment the question classifier 448 may be configured toidentify whether an incoming question is within a limited number ofapproved topics such as science, history, and/or geography. If thequestion is not within the limited number of approved topics, the systemmay determine the question is not suitable (508:No) for search-basedanswering. The search acceptability filter 440 may also include othercomponents for considering question suitability.

If the question is suitable (508:Yes) the system may send (146) the textof the query to a search engine 480. The search engine 480 may searchdata source(s) 180 to determine one or more data sets that may beresponsive to the query text data. The search engine 480 may then sendURLs or other links to those data sets to the search manager 410, whichreceives (148) them. The search answer controller 420 may then obtain(150) data corresponding to those links and send that data to the searchresult manager 450. The search results manager may then determine (152)answer data from one of the top search results. For example, once thequery is determined to be suitable, the search answer controller 420which then sends a request to an API of the search engine 480 to run asearch using the text data. The top certain number (for example 3) linkscorresponding to the search results get sent to the search answercontroller 420. The links may point to one or more data source(s) 180.The search answer controller 420 may then use the links to access thedata source(s) 180 (for example, Wikipedia pages, other web pages, datastorage corresponding to a user device, etc.) and obtain the data linkedthereto or, alternatively, to obtain the first few paragraphs orsentences of data that is indicated by the link(s). The search answercontroller 420 and/or search result manager 450 may then isolate aportion of the answer data corresponding to a top search result. Thatportion may correspond, for example, to one or two sentences from a datasource 180. The system may then determine (512) if the search resultanswer of the portion is appropriate as an answer to the originalquestion. To do this the answer relevance classifier 460 may input theanswer portion as well as the original query text data (and potentiallyother data such as feature data corresponding to the answer portion,original query text data, etc.) and may process those inputs using amachine learning model to determine a score as to whether the answerportion corresponds to the original question being asked. If yes (e.g.,if the score is above a score threshold), the system may determine thatthe answer is responsive to the user's question (512:Yes).

If the answer portion is not responsive to the user's question (512:No),the system may again look through the search result data to find adifferent portion of answer data potentially responsive to the user'squestion until suitable answer data is found that satisfies the answerrelevance classifier 460. If no such data is found, processing of thetext data by the search manager 410 may be discontinued (510) and thetext data may be sent or otherwise indicated for alternate processing.

If an answer portion is responsive to the user's question (512:Yes) thesystem may determine (522) a source indicator from the appropriateanswer. For example, if the answer portion is taken from “Website X,”text corresponding to Website X may be included in the source indicator.Turning again to FIG. 5B, if an answer is not found in the knowledgebase (528:No) and an answer is found using the search-based approach,the system may combine (154) the appropriate search results answer datawith the source indicator to create output data. The system may thenperform (156) TTS on the output data to obtain output audio data and maysend (536) the output audio data to a local device so the local devicemay output speech corresponding to the answer to the question. Forexample, the system may respond to the user's query with synthesizedspeech such as “I found an answer to your question. According to[website X]: [answer data].”

As noted above, the system (suing feature component 404) may determine(506) feature values corresponding to the question and may alsoconfigure a feature vector representing those values. The feature vectorrepresenting the question may then be used by the described models todetermine (508) if the question is appropriate for answering during aweb search, such as using components discussed above, such as thosediscussed in reference to FIG. 4. Feature values may include variousdata points representing characteristics of the text. Example featuresinclude words of the text, word embeddings for the words of the text,parts of speech tags for the words of the text, indicators for entitiesnamed in the text, or other features.

One feature that may be determined is any subject-verb-object (SVO) orsubject-relation-object triples included in the particular text data. Atriple is three expressions that give some sense of the syntacticstructure of the text. Each expression may be a single word or multiplewords. For example, a subject may have several words that identify thesubject entity. Identifying a triple in a text segment may be performedusing known semantic analysis techniques, such as those currently usedto parse questions for querying a knowledge base. While a triple may notbe enough to the entire semantic content of a text segment, the triplemay impart some information that can be useful in identifying a textsegment to answer a particular question. Any triple(s) identified for aparticular text segment may be indicated in the feature data. Onefeature may include entity and relation mentions. Other features thatmay be included for a question are any keywords included in a textsegment (e.g., the words of the text data). For example, words that maybe useful in determining the subject of a text segment may be identifiedand indicated in feature data. Various known techniques for identifyingkeywords within a text segment may be used. Another feature that may bedetermined is information about the source of the question (such as theidentity of a user submitting the question, a device from which thequestion originated, etc.). Other feature data may also be used.

The system may also include a formatter and/or formatter model that maybe configured to operate on many different question types and may beparticularly configured to output data in a voice-controlled system,where the eventual output to the user is known a priori to besynthesized speech. Thus the formatter may remove extraneous text fromthe text segment and make the answer more efficient and more pleasingfor a user of a voice-controlled question answering system. Theformatter and/or formatter model may attempt to create answer data withsome syntactic similarity to the question and may even insert data notincluded in the text segment (such as the user's name or otherinformation) to make the answer experience pleasing to the user.

Thus the present system offers a method for answering user questions ina voice-controlled environment that improves upon existing web searchtechnology, which simply returns a list of links unsuitable forvoice-based exchanges. The present system is also more robust than aknowledge base system alone, as such systems are limited in theinformation they can provide. Although illustrated as being used with avoice-controlled question answering system, it may also be used with anon-voice controlled system.

Various machine learning techniques may be used to perform the trainingof the various classifiers and models discussed above. Models may betrained and operated according to various machine learning techniques.Such techniques may include, for example, neural networks (such as deepneural networks and/or recurrent neural networks), inference engines,trained classifiers, etc. Examples of trained classifiers includeconditional random fields (CRF) classifiers, Support Vector Machines(SVMs), neural networks (such as deep neural networks and/or recurrentneural networks), decision trees, AdaBoost (short for “AdaptiveBoosting”) combined with decision trees, and random forests. Focusing onSVM as an example, SVM is a supervised learning model with associatedlearning algorithms that analyze data and recognize patterns in thedata, and which are commonly used for classification and regressionanalysis. Given a set of training examples, each marked as belonging toone of two categories, an SVM training algorithm builds a model thatassigns new examples into one category or the other, making it anon-probabilistic binary linear classifier. More complex SVM models maybe built with the training set identifying more than two categories,with the SVM determining which category is most similar to input data.An SVM model may be mapped so that the examples of the separatecategories are divided by clear gaps. New examples are then mapped intothat same space and predicted to belong to a category based on whichside of the gaps they fall on. As another example, CRF is a class ofstatistical models used for structured predictions. In particular, CRFsare a type of discriminative undirected probabilistic graphical models.A CRF can predict a class label for a sample while taking into accountcontextual information for the sample. CRFs may be used to encode knownrelationships between observations and construct consistentinterpretations. A CRF model may thus be used to label or parse certainsequential data, like query text data as described above. Classifiersmay issue a “score” indicating which category the data most closelymatches. The score may provide an indication of how closely the datamatches the category.

In order to apply the machine learning techniques, the machine learningprocesses themselves need to be trained. Training a machine learningcomponent such as, in this case, one of the first or second models,requires establishing a “ground truth” for the training examples. Inmachine learning, the term “ground truth” refers to the accuracy of atraining set's classification for supervised learning techniques.Various techniques may be used to train the models includingbackpropagation, statistical learning, supervised learning,semi-supervised learning, stochastic learning, or other knowntechniques. Many different training examples may be used to perform thetraining discussed herein.

FIG. 6 is a block diagram conceptually illustrating a local device 110that may be used with the described system. FIG. 7 is a block diagramconceptually illustrating example components of a remote device, such asa remote server 120 that may assist with ASR, NLU processing, or commandprocessing. Multiple such servers 120 may be included in the system,such as one server(s) 120 for training ASR models, one server(s) forperforming ASR, one server(s) 120 for performing NLU, etc. In operation,each of these devices (or groups of devices) may includecomputer-readable and computer-executable instructions that reside onthe respective device (110/120), as will be discussed further below.

Each of these devices (110/120) may include one or morecontrollers/processors (604/704), that may each include a centralprocessing unit (CPU) for processing data and computer-readableinstructions, and a memory (606/706) for storing data and instructionsof the respective device. The memories (606/706) may individuallyinclude volatile random access memory (RAM), non-volatile read onlymemory (ROM), non-volatile magnetoresistive (MRAM) and/or other types ofmemory. Each device may also include a data storage component (608/708),for storing data and controller/processor-executable instructions. Eachdata storage component may individually include one or more non-volatilestorage types such as magnetic storage, optical storage, solid-statestorage, etc. Each device may also be connected to removable or externalnon-volatile memory and/or storage (such as a removable memory card,memory key drive, networked storage, etc.) through respectiveinput/output device interfaces (602/702).

Computer instructions for operating each device (110/120) and itsvarious components may be executed by the respective device'scontroller(s)/processor(s) (604/704), using the memory (606/706) astemporary “working” storage at runtime. A device's computer instructionsmay be stored in a non-transitory manner in non-volatile memory(606/706), storage (608/708), or an external device(s). Alternatively,some or all of the executable instructions may be embedded in hardwareor firmware on the respective device in addition to or instead ofsoftware.

Each device (110/120) includes input/output device interfaces (602/702).A variety of components may be connected through the input/output deviceinterfaces, as will be discussed further below. Additionally, eachdevice (110/120) may include an address/data bus (624/724) for conveyingdata among components of the respective device. Each component within adevice (110/120) may also be directly connected to other components inaddition to (or instead of) being connected to other components acrossthe bus (624/724).

Referring to the device 110 of FIG. 6, the device 110 may include adisplay 618, which may comprise a touch interface 619. Or the device 110may be “headless” and may primarily rely on spoken commands for input.As a way of indicating to a user that a connection between anotherdevice has been opened, the device 110 may be configured with a visualindicator, such as an LED or similar component (not illustrated), thatmay change color, flash, or otherwise provide visual indications by thedevice 110. The device 110 may also include input/output deviceinterfaces 602 that connect to a variety of components such as an audiooutput component such as a speaker 660, a wired headset or a wirelessheadset (not illustrated) or other component capable of outputtingaudio. The device 110 may also include an audio capture component. Theaudio capture component may be, for example, a microphone 650 or arrayof microphones, a wired headset or a wireless headset (not illustrated),etc. The microphone 650 may be configured to capture audio. If an arrayof microphones is included, approximate distance to a sound's point oforigin may be performed acoustic localization based on time andamplitude differences between sounds captured by different microphonesof the array. The device 110 (using microphone 650, wakeword detectionmodule 220, ASR module 250, etc.) may be configured to determine audiodata corresponding to detected audio data. The device 110 (usinginput/output device interfaces 602, antenna 614, etc.) may also beconfigured to transmit the audio data to server 120 for furtherprocessing or to process the data using internal components such as awakeword detection module 220.

For example, via the antenna(s), the input/output device interfaces 602may connect to one or more networks 199 via a wireless local areanetwork (WLAN) (such as WiFi) radio, Bluetooth, and/or wireless networkradio, such as a radio capable of communication with a wirelesscommunication network such as a Long Term Evolution (LTE) network, WiMAXnetwork, 3G network, etc. A wired connection such as Ethernet may alsobe supported. Through the network(s) 199, the speech processing systemmay be distributed across a networked environment.

The device 110 and/or server 120 may include an ASR module 250. The ASRmodule in device 110 may be of limited or extended capabilities. The ASRmodule 250 may include the language models 254 stored in ASR modelstorage component 252, and an ASR module 250 that performs the automaticspeech recognition process. If limited speech recognition is included,the ASR module 250 may be configured to identify a limited number ofwords, such as keywords detected by the device, whereas extended speechrecognition may be configured to recognize a much larger range of words.

The device 110 and/or server 120 may include a limited or extended NLUmodule 260. The NLU module in device 110 may be of limited or extendedcapabilities. The NLU module 260 may comprise the name entityrecognition module 262, the intent classification module 264 and/orother components. The NLU module 260 may also include a stored knowledgebase and/or entity library, or those storages may be separately located.

The device 110 and/or server 120 may also include an application 290that is configured to execute commands/functions associated with aspoken command as described above.

The server 120 (and potentially the device 110 as well) may also includea TTS module 214 to process text into speech for use with the voicecontrolled interface or other features of the system 100.

The device 110 may include a wakeword detection module 220, which may bea separate component or may be included in an ASR module 250. Thewakeword detection module 220 receives audio signals and detectsoccurrences of a particular expression (such as a configured keyword) inthe audio. This may include detecting a change in frequencies over aspecific period of time where the change in frequencies results in aspecific audio signature that the system recognizes as corresponding tothe keyword. Keyword detection may include analyzing individualdirectional audio signals, such as those processed post-beamforming ifapplicable. Other techniques known in the art of keyword detection (alsoknown as keyword spotting) may also be used. In some embodiments, thedevice 110 may be configured collectively to identify a set of thedirectional audio signals in which the wake expression is detected or inwhich the wake expression is likely to have occurred.

The wakeword detection module 220 receives captured audio and processesthe audio (for example, using model(s) 232) to determine whether theaudio corresponds to particular keywords recognizable by the device 110and/or system 100. The storage 608 may store data relating to keywordsand functions to enable the wakeword detection module 220 to perform thealgorithms and methods described above. The locally stored speech modelsmay be pre-configured based on known information, prior to the device110 being configured to access the network by the user. For example, themodels may be language and/or accent specific to a region where the userdevice is shipped or predicted to be located, or to the userhimself/herself, based on a user profile, etc. In an aspect, the modelsmay be pre-trained using speech or audio data of the user from anotherdevice. For example, the user may own another user device that the useroperates via spoken commands, and this speech data may be associatedwith a user profile. The speech data from the other user device may thenbe leveraged and used to train the locally stored speech models of thedevice 110 prior to the user device 110 being delivered to the user orconfigured to access the network by the user. The wakeword detectionmodule 220 may access the storage 608 and compare the captured audio tothe stored models and audio sequences using audio comparison, patternrecognition, keyword spotting, audio signature, and/or other audioprocessing techniques.

The server may include a model training component. The model trainingcomponent may be used to train classifier(s)/models discussed above.

The server may also include a search manager component 410 that canoperate to answer queries using the search-based methods disclosed here.The search manager component 410 may include a search answer controller420, search acceptability filter 440, search result manager 450, answerrelevance classifier 460 and storage which may store the machine trainedmodel(s)/classifier(s) used to operate the present system, including butnot limited to blacklist manager 442, question type manager 444,suitability manager 446, question classifier 448 and/or answer relevanceclassifier 460.

As noted above, multiple devices may be employed in a single speechprocessing system. In such a multi-device system, each of the devicesmay include different components for performing different aspects of thespeech processing. The multiple devices may include overlappingcomponents. The components of the devices 110 and server 120, asillustrated in FIGS. 6 and 7, are exemplary, and may be located astand-alone device or may be included, in whole or in part, as acomponent of a larger device or system.

As illustrated in FIG. 8 multiple devices (120, 120 x, 110 a to 110 f)may contain components of the system 100 and the devices may beconnected over a network 199. Network 199 may include a local or privatenetwork or may include a wide network such as the internet. Devices maybe connected to the network 199 through either wired or wirelessconnections. For example, a speech controlled device 110 a, a tabletcomputer 110 b, a smart phone 110 c, a refrigerator 110 d, a smart watch110 e, and/or a vehicle 110 f may be connected to the network 199through a wireless service provider, over a WiFi or cellular networkconnection or the like. Other devices are included as network-connectedsupport devices, such as a server 120, application developer devices 120x, or others. The support devices may connect to the network 199 througha wired connection or wireless connection. Networked devices 110 maycapture audio using one-or-more built-in or connected microphones 650 oraudio capture devices, with processing performed by ASR, NLU, or othercomponents of the same device or another device connected via network199, such as an ASR 250, NLU 260, etc. of one or more servers 120.

The concepts disclosed herein may be applied within a number ofdifferent devices and computer systems, including, for example,general-purpose computing systems, speech processing systems, anddistributed computing environments.

The above aspects of the present disclosure are meant to beillustrative. They were chosen to explain the principles and applicationof the disclosure and are not intended to be exhaustive or to limit thedisclosure. Many modifications and variations of the disclosed aspectsmay be apparent to those of skill in the art. Persons having ordinaryskill in the field of computers and speech processing should recognizethat components and process steps described herein may beinterchangeable with other components or steps, or combinations ofcomponents or steps, and still achieve the benefits and advantages ofthe present disclosure. Moreover, it should be apparent to one skilledin the art, that the disclosure may be practiced without some or all ofthe specific details and steps disclosed herein.

Aspects of the disclosed system may be implemented as a computer methodor as an article of manufacture such as a memory device ornon-transitory computer readable storage medium. The computer readablestorage medium may be readable by a computer and may compriseinstructions for causing a computer or other device to perform processesdescribed in the present disclosure. The computer readable storage mediamay be implemented by a volatile computer memory, non-volatile computermemory, hard drive, solid-state memory, flash drive, removable diskand/or other media. In addition, components of one or more of themodules and engines may be implemented as in firmware or hardware, suchas the acoustic front end 256, which comprise among other things, analogand/or digital filters (e.g., filters configured as firmware to adigital signal processor (DSP)).

As used in this disclosure, the term “a” or “one” may include one ormore items unless specifically stated otherwise. Further, the phrase“based on” is intended to mean “based at least in part on” unlessspecifically stated otherwise.

1-20. (canceled)
 21. A computer-implemented method, comprising:receiving input data corresponding to a user input; determining theinput data corresponds to an intent to receive information responsive toa query; determining, using the knowledge base, first data correspondingto a potential answer to the query; determining the first data fails tocorrespond to an answer to the query; sending data representing thequery to a search component; receiving, from the search component, textdata; and at least partially in response to determining that the firstdata fails to correspond to the answer to the query, causing audio datarepresenting the text data to be output in response to the query. 22.The computer-implemented method of claim 21, further comprising:determining a source corresponding to the text data; and determining theaudio data to include a representation of an indicator corresponding tothe source.
 23. The computer-implemented method of claim 21, furthercomprising: performing natural language understanding (NLU) on the inputdata to determine NLU result data; and processing the NLU result data todetermine the input data corresponds to the intent to receiveinformation responsive to a query.
 24. The computer-implemented methodof claim 23, further comprising: receiving input audio datacorresponding to the user input; and performing automatic speechrecognition (ASR) on the input audio data to determine ASR resultscomprising the input data.
 25. The computer-implemented method of claim21, further comprising: processing the input data to determine a firstportion corresponding to the query; and determining the datarepresenting the query based at least in part on the first portion. 26.The computer-implemented method of claim 21, wherein: the first dataincludes first confidence data corresponding to the knowledge base andthe query; determining that the first data fails to correspond to ananswer to the query is based at least in part on the first confidencedata.
 27. The computer-implemented method of claim 26, furthercomprising: determining the first confidence data fails to satisfy athreshold.
 28. The computer-implemented method of claim 26, furthercomprising: determining second confidence data corresponding to the textdata, wherein determining that the first data fails to correspond to ananswer to the query is further based at least in part on the secondconfidence data.
 29. The computer-implemented method of claim 21,wherein: sending the data representing the query to the search componentis performed at least partially in response to determining that thefirst data fails to correspond to an answer to the query.
 30. Thecomputer-implemented method of claim 21, further comprising: processingthe text data using a text-to-speech component to determine the audiodata.
 31. A system comprising: at least one processor; and at least onememory comprising instructions that, when executed by the at least oneprocessor, cause the system to: receive input data corresponding to auser input; determine the input data corresponds to an intent to receiveinformation responsive to a query; determine, using the knowledge base,first data corresponding to a potential answer to the query; determinethe first data fails to correspond to an answer to the query; send datarepresenting the query to a search component; receive, from the searchcomponent, text data; and at least partially in response to determiningthat the first data fails to correspond to the answer to the query,cause audio data representing the text data to be output in response tothe query.
 32. The system of claim 31, wherein the at least one memoryfurther comprises instructions that, when executed by the at least oneprocessor, further cause the system to: determine a source correspondingto the text data; and determine the audio data to include arepresentation of an indicator corresponding to the source.
 33. Thesystem of claim 31, wherein the at least one memory further comprisesinstructions that, when executed by the at least one processor, furthercause the system to: perform natural language understanding (NLU) on theinput data to determine NLU result data; and process the NLU result datato determine the input data corresponds to the intent to receiveinformation responsive to a query.
 34. The system of claim 33, whereinthe at least one memory further comprises instructions that, whenexecuted by the at least one processor, further cause the system to:receive input audio data corresponding to the user input; and performautomatic speech recognition (ASR) on the input audio data to determineASR results comprising the input data.
 35. The system of claim 31,wherein the at least one memory further comprises instructions that,when executed by the at least one processor, further cause the systemto: process the input data to determine a first portion corresponding tothe query; and determine the data representing the query based at leastin part on the first portion.
 36. The system of claim 31, wherein: thefirst data includes first confidence data corresponding to the knowledgebase and the query; and the instructions that cause the system todetermine that the first data fails to correspond to an answer to thequery are based at least in part on the first confidence data.
 37. Thesystem of claim 36, wherein the at least one memory further comprisesinstructions that, when executed by the at least one processor, furthercause the system to: determine the first confidence data fails tosatisfy a threshold.
 38. The system of claim 36, wherein the at leastone memory further comprises instructions that, when executed by the atleast one processor, further cause the system to: determine secondconfidence data corresponding to the text data, wherein the instructionsthat cause the system to determine that the first data fails tocorrespond to an answer to the query are further based at least in parton the second confidence data.
 39. The system of claim 31, wherein theinstructions that cause the system to send the data representing thequery to the search component are executed at least partially inresponse to determining that the first data fails to correspond to ananswer to the query.
 40. The system of claim 31, wherein the at leastone memory further comprises instructions that, when executed by the atleast one processor, further cause the system to: process the text datausing a text-to-speech component to determine the audio data.